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Background 

Field of the Invention 

[0003] The present invention relates generally to document printers and, more 
specifically, to document printers that can receive, process, and transform multimedia 
data from a peripheral device. 
Background of the Invention 

[0004] Cost and quality improvements in multimedia technologies have led to a 
proliferation of digital devices with multimedia capabilities. High-quality video cameras 
and cellular phones with multimedia capabilities are becoming commonplace in the home 
and workplace, and have proven to be useful for diverse purposes ranging from 
teleconferencing to managing information. Multimedia data captured by such devices are 
typically delivered in an unprocessed form to a medium such as a digital tape or memory 
card. 

[0005] A conventional printer can receive multimedia data in a number of formats 
and then print the contents of those data in accordance with the proper format. But while 
conventional printers can print documents in a wide variety of formats, these printers are 
fundamentally limited in their ability to take in contents from multimedia devices such as 
video cameras and cellular phones and process the data to create a useable record. For 
example, it is standard technology for a printer to produce images of static text, pictures, 
or a combination of the two. However, creating multimedia output from a peripheral 
device typically involves several disparate steps, each potentially requiring considerable 
effort. Oftentimes, a user will have to convert and transfer multimedia data in different 
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stages to different devices - for instance transferring a video clip from a video camera, 
then into a summary file containing excerpts of the video file, then to a memory or output 
device. While the processing of a multimedia file from a peripheral device commonly 
involves the same repeated tasks, there is no easy way to automate them. In addition, 
because the multimedia data from a peripheral device are typically not printed to a paper 
document, they are difficult to incorporate into the existing paper-based workflow by 
which most offices function. Although means do exist to map multimedia data from a 
peripheral device to paper-friendly and electronic outputs, these additional conversion 
steps are often not automated or performed. 

[0006] Thus, there is a need for an integrated printer that can receive multimedia 
data from a peripheral device, process it, and output a printed document or other media. 
It is further desirable that such a printer be able to perform at least some of the necessary 
processing itself, while some of the processing may be performed on an external device, 
rather than require an attached computer or other device to perform all of the processing. 

Summary of the Invention 
[0007] The present invention overcomes the deficiencies and limitations of the prior 
art by providing systems and methods in which multimedia data from a peripheral device 
are received by a printer, the data are processed, and the printer outputs the result. In a 
typical hardware configuration for such a multifunction printer, a printer includes a 
peripheral interface that communicates with a peripheral device, a print engine that 
produces a paper or other printed output, and one or more electronic devices that produce 
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a related electronic output. Together, the printed and electronic outputs provide an 
improved representation of the multimedia data from the peripheral device over that of a 
convention paper printer. 

[0008] In one embodiment, a printing system includes a network coupled to a 
network device and a printing system. An interface receives multimedia data from a 
peripheral device, and a multimedia processing system is coupled to the interface for 
processing the multimedia data. Based on any of a number of desired applications, the 
multimedia processing system determines a printed representation of the multimedia data 
and an electronic representation of the multimedia data. To enable sharing of the 
computing load, the multimedia processing system may reside at least in part on the 
printing system and at least in part on the peripheral device. A printed output system in 
communication with the multimedia processing system receives the printed representation 
and produces a corresponding printed output. Similarly, an electronic output system in 
communication with the multimedia processing system receives the electronic 
representation and produces a corresponding electronic output. In this way, the printer 
creates a representation of multimedia data from a peripheral device by producing a 
printed output and an electronic output. 

[0009] In various embodiments, the system includes hardware and software for 
processing multimedia contents and various mechanisms for creating the electronic and 
printed outputs. For example, the interface may include a single communication 
interface, a network interface, a wireless interface, personal digital assistant (PDA) 
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device, a cellular phone, a removable media storage device reader, a video input device 
(such as a DVD reader), an audio input device (such as a CD reader or an MP3 player), a 
digital video recorder (e.g., TiVO), screen capture hardware, a video and/or audio 
recorder, or any of a number of different types of devices that can receive multimedia 
data. Similarly, the electronic output system may write the electronic representation to 
one or more different types of removable media storage devices, such as a DVD, a digital 
video recorder, a digital audio recorder, a CD, a flash card, a computer disk, an SD disk, 
or another computer-readable medium. The electronic output system may also include a 
disposable media writer, a self-destructing media writer, a video display, an audio 
speaker, a driver for a speaker system (such as an embedded MIDI player), or an 
embedded web page display. In this way, a multifunction printer can be configured to 
process any of a large number of multimedia data from various peripheral devices, 
allowing various embodiments of the printer to meet the needs of many different 
applications. 

[0010] Because of the great many combinations of input and output devices 
possible for the printing system, the system may include hardware, software, or a 
combination thereof for performing a wide variety of different operations on the 
multimedia data. In this way, the system can be configured to produce various types of 
printed and electronic outputs based on received multimedia data to meet the needs of 
different applications. To solve various problems, in embodiments of the system, the 
multimedia processing system includes one or more embedded multimedia modules. 
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[0011] Moreover, the processing logic in the printing system can be configured to 
communicate with a peripheral device via the peripheral interface. Thus, the processing 
logic has the capability to operate the peripheral device and transfer data to the peripheral 
device. 

[0012] These different tasks may be performed on the printer by the multimedia 
processing system, or partially on the printer by the multimedia processing system in 
conjunction with one or more electronic devices capable of performing some of the 
required processing steps. The printer can thus balance the required processing of the 
multimedia data between the printer and one or more connected electronic devices, such 
as a personal computer or an external network service. By conducting at least some of 
the processing on the printer, the printer relieves at least some of the processing load on 
external devices that the printer's additional functionality may require. 
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Brief Description of the Drawings 
[0013] FIG. 1 is a schematic diagram of a system in accordance with an 
embodiment of the invention. 

[0014] FIG. 2 is a flow diagram of the operation of the system in accordance with 
an embodiment of the invention. 

[0015] FIG. 3 is a schematic diagram of various peripheral interfaces of the printer, 
in accordance with embodiments of the invention. 

[0016] FIG. 4 is a schematic diagram of the printer's printed output system, in 
accordance with an embodiment of the invention. 

[0017] FIG. 5 is a schematic diagram of various electronic media output systems of 
the printer, in accordance with embodiments of the invention. 

[0018] FIG. 6 is a schematic diagram of the multimedia module of the printer, in 
accordance an embodiment of the invention. 

[0019] FIG. 7 is a flow diagram of the operation of the system for automatically 
receiving multimedia data from a docked peripheral device, in accordance to an 
embodiment of the invention. 

[0020] FIG. 8 is an illustration of interactive communication with the printer in 
accordance with an embodiment of the invention. 
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Detailed Description of the Preferred Embodiments 
[0021] Various embodiments of a printing system with a peripheral interface enable 
the printing of multimedia in a useful and intelligent format. To create a representation of 
this multimedia data, the printing system produces a printed output and a related 
electronic output, which together provide a representation of the received multimedia 
data. Depending on the desired application, the printing system includes a multimedia 
module with any number of functionalities for processing the multimedia data, printing 
the printed output, and producing the electronic output. Therefore, a number of 
embodiments of the printing system are described herein to show how such a system can 
be configured in a virtually limitless number of combinations to solve or address a great 
number of needs that exist. 

System Architecture 

[0022] FIG. 1 is a high-level diagram of one embodiment of a printing system, 
which includes a printer 100, a network 155, and a network device 170. The network 155 
and the network device 170 are optional components to the printer 100. In other 
embodiments, the printer 100 can be configured to operate as a standalone printer without 
a network connection. Printer 100 includes a peripheral interface 105, user interfaces 
1 10a and 1 10b, a printed output system 1 15, an electronic output system 120, and a 
multimedia processing system 125. Capable of receiving multimedia data from a 
peripheral device 151, the peripheral interface 105 can take a variety of forms and may 
include one or more devices that can receive multimedia data or create multimedia data 
by observing a media event. Similarly, the printed output system 1 1 5 and the electronic 
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output system 120 can take a variety of forms and may each include one or more devices 
that can produce, respectively, a printed output 165 and an electronic output 175. 

[0023] In one embodiment, the multimedia processing system 125 includes a 
memory 130, a processor 135, and a multimedia processing module 140. The multimedia 
processing module 140, which is described in more detail below, may include software, 
hardware, or a combination thereof for implementing at least a portion of the 
functionality of the multifunction printer 100. The multimedia processing system 125 is 
coupled to the peripheral interface 105, allowing it to communicate therewith. The media 
processing system 125 is also coupled to the printed output system 115 and to the 
electronic output system 120 for providing the appropriate commands and data to those 
systems. 

[0024] In an embodiment, the printer 1 00 may further include a network interface 
150, functionally coupled to the multimedia processing system 125. The network 
interface 150 allows the printer 100 to communicate with other electronic devices, such 
as network device 170 and external service 160. In one embodiment, the network device 
170 is a computer system, such as a personal computer. Beneficially, the network device 
170 includes processing capability for performing any desired processing on the media 
data. In this way, the network device 1 70 can relieve the printer 100 of some of the 
processing load required to produce printed and electronic outputs from the multimedia 
data. In one embodiment, the network device 170 includes a user interface 1 10b that 
allows a user to make selections about the processing of the multimedia and/or about the 
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format or destination of the printed or electronic outputs. In other embodiments, the user 
interface 1 10b can be located on another attached device, or the user interface 1 10a can 
be located on the printer 100 itself The user interface 110 may include a display system 
and software for communicating with an attached display. For example, a user may 
communicate with the printing system via user interface 1 10 to configure output layouts 
for a multimedia presentation. 

[0025] In another embodiment, the printer 100 is coupled to an external service 
160, which includes hardware and/or software for performing some of the processing 
tasks on the multimedia to be printed. In a typical embodiment, a remote service provider 
operates the external service 160. In such an embodiment, whereas the network device 
1 70 may communicate with the printer 1 00 over a local area network, the external service 
may communicate with the printer 100 over a wide area network or over the Internet. By 
sharing the media processing tasks with an external service 160, possibly operated by a 
service provider, the printing system can perform tasks that are under the control of the 
service provider. In this way, a service can be set up around a particular form of 
multimedia processing, where a user pays for use of the service. 

. [0026] FIG. 2 shows an overview of a generalized process in which the printer 100 
creates a representation of multimedia data, in accordance with one embodiment of the 
invention. The printer 100 first receives 205 multimedia data from a peripheral device 
151 via the peripheral interface 105. This multimedia data may be received as digital 
data, or it may be an observable event that the interface 105 records as digital data. 
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Coupled to the interface 105 to receive the multimedia data, the multimedia processing 
system 125 processes 210 the data to generate printed and electronic outputs. This 
processing 210 is performed in accordance with the intended functionality of the printer 
100, and examples of different operations are described in greater detail below. 

[0027] In one embodiment, the system automatically generates a printed output or 
an electronic output of the multimedia data based on a predefined format and the 
functionality of the peripheral device 151 that is connected to the peripheral interface 105. 
For example, the multimedia processing system can take a video clip from a digital video 
camera connected to peripheral interface 105 and automatically generate a printed output 
on a video paper containing key frames and bar codes for accessing video segments of the 
video file. 

[0028] In another embodiment, the system includes a user interface 1 10 to allow a 
user to preview the generated outputs. If the user desires to refine the processing 215, the 
user can enter commands, which the printer 100 receives 220 by way of the user interface 
110. Based on the user's commands, the printer 100 then repeats the processing 210 of 
the media data to generate new printed and electronic outputs. This refinement process 
can be iterated until the user is satisfied with the printed and electronic outputs. When 
this occurs, the user will indicate that the printing should commence, for example, by 
invoking a print command with the user interface 110. The multimedia processing 
system 125 then sends the generated printed output to the printed output system 115 and 
the electronic output to the electronic output system 120. The printed output system 115 
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then creates 225 a printed output 165, and the electronic output system 120 creates 230 an 
electronic output 175. 

[0029] It can be appreciated that this generalized description of a multifunction 
printer lends itself to a great number of specific configurations and applications. 
Accordingly, examples of the possible configurations, applications, and particular 
components are further described. 

Peripheral Interface 

[0030] The peripheral interface 105 can be designed to accommodate any suitable 
type of multimedia peripheral devices 151. Because of the great variety of types and 
formats of multimedia data, the peripheral interface 105 may take any number of forms to 
accept any devices 151 that a user might wish to connect and print the multimedia 
contents from the devices. FIG. 3 illustrates some examples of different interfaces 105 by 
which the printer 1 00 can receive multimedia data from a peripheral device 1 5 1 . In 
particular implementations, the printer 100 may have only one or only a subset of these 
types of interfaces 105, and in addition the printer 100 may have other types of interfaces 
not shown. 

[0031] As shown in FIG. 3, the printer 1 00 may include a communication interface 
305 that allows the printer 100 to be communicatively coupled to another electronic 
device. Depending on the desired input, the interface 305 may allow the computer to 
communicate with a wide variety of different peripheral devices 151 that can provide the 
printer 100 multimedia data to print. Without intending to limit the types of devices, the 
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interface 305 may allow the printer 100 to received media data from peripheral devices 
151 such as computer systems, computer networks, digital cameras, cellular telephones, 
PDA devices, video cameras, media Tenderers (such as DVD and CD players), media 
receivers (such as televisions, satellite receivers, set-top boxes, and the like), digital video 
recorders (such as a TiVO), a portable meeting recorder, external storage devices, video 
game systems, or any combination thereof. The connection type for the interface 305 can 
take a variety of forms based on the type of device that is intended to be connected to the 
printer 100 and the available standard connections for that type of device. For example, 
the interface 305 may comprise a port for connecting the device using a connection type 
such as USB, serial, FireWire, SCSI, IDE, RJ1 1, parallel port (e.g., bi-directional, 
Enhanced Parallel Port (EPP), Extended Capability Port (ECP), IEEE 1284 Standard 
parallel port), optical, composite video, component video, or S-video, or any other 
suitable connection type. 

[0032] In another embodiment, the printer 100 includes a wireless interface 310. 
As illustrated, the wireless interface 310 allows the printer 100 to receive multimedia data 
from a wireless peripheral device 151 external to the printer 100. The wireless interface 
310 may allow the printer 100 to communicate with any number of wireless 
communication systems, such as wireless components on a home or business network, 
cellular phones and other portable wireless devices, satellites, satellite dishes, and devices 
using radio transmissions. Depending on the types of external devices with which the 
printer 100 is desired to communicate, the wireless interface 310 may comprise hardware 
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and/or software that implements a wireless communications protocol, such as that 
described in IEEE 802.1 1 or the Bluetooth standard. 



[0033] In another embodiment, the printer 100 receives media data from a 
removable media storage reader 3 1 5 that is built into the printer 1 00. The removable 
media storage reader 315 may be configured to accommodate any type of removable 
media storage device, such as DVDs, CDs, video cassette tapes, audio cassette tapes, 
floppy disks, ZIP disks, flash cards, micro-drives, memory sticks, SD disks, or any other 
suitable type of multimedia storage devices. Moreover, the printer 100 may have a 
plurality of removable multimedia storage readers 315 to accommodate multiple types of 
media storage devices. 

[0034] In another embodiment, the printer 100 includes a docking station 335 that is 
built into the printer 100. The docking station 335 may be configured to accommodate 
any type of peripheral device, such cell phones, digital audio recorders, video camcorders, 
portable meeting recorders, fixed position meeting recorders, head-mounted video 
cameras, office-based PC experience capture systems, or any other suitable type of 
multimedia peripheral devices. Moreover, the printer 100 may have a plurality of 
docking stations 335 to accommodate multiple types of peripheral devices. Furthermore, 
it will be understood that a peripheral device 151 may also be communicatively coupled 
to the communication interface 305 via an external docking station. 

[0035] In another embodiment, the printer may include video capture hardware 355. 
In one embodiment, the video capture hardware 355 is designed to be coupled to a 
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computing system by a video cable thereof. The video cable from a display is attached to 
the printer 100, where the video signal is split with one signal directed to the computing 
system and another signal to the video capture hardware 355. The video capture 
hardware 355 performs a differencing between successive frames of the video signal and 
saves frames with a difference that exceeds a threshold on a secondary storage in the 
printer 100. This offloads such processing from the computing system, thereby 
improving responsiveness and user experience and providing an easily browseable record 
of a user's activities during the day. To take advantage of the printing capabilities of the 
multifunction printer, the user can choose to print selected frames captured by the video 
capture hardware 355. The printing can be generated on demand with the user interface 
1 10 on the printer or from the attached computing system, or automatically with 
scheduling software. In this way, a user can view a replay of any actions taken on the 
computing system. Notably, the captured content can be effectively compressed because 
the differences between frames are small. 

[0036] In another embodiment, the video capture hardware 355 is coupled to a 
converter module 360, such as VGA-to-NTSC conversion hardware. Such an 
embodiment could be used in conjunction with a projector to capture presentations made 
with the projector. Audio capture could also be employed to record a speaker's oral 
presentation. To use the video capture hardware 355 in this way, a user could connect a 
laptop or other computing system and the projector to the printer 100. The printer 100 
then captures video frames and compares them to the most recently captured frame and 
retains those frames that are different. A parallel audio track may also be saved. This 
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capability could also be used in a desktop printer to record a presentation made on a 
computing system connected to the printer. The printer can then serve the audio itself or 
it can be written to a digital medium, such as an SD disk that can be played from a cell 
phone or a PDA. The audio could also be written to a bar code on a printed 
representation. 

Printed Output System 

[0037] The printed output system 115 may comprise any standard printing 
hardware, including that found in standard laser printers, inkjet printers, thermal wax 
transfer printers, dye sublimation printers, dot matrix printers, plotters, or any other type 
of printing mechanisms suitable for creating a printer image on an appropriate physical 
medium. In the example described herein, a laser printer mechanism is described; 
however, it should be understood that any suitable printing system can be used. The 
printer 100 includes any necessary subsystems, as know by one skilled in the art, to print 
on a printable medium, such as a sheet of paper. 

[0038] In one embodiment, the printed output system 115 comprises a media supply 
handler 405 that receives blank paper to be printed on. The media supply handler 405 
typically obtains the paper from a supply tray 410. The printer 100 may include multiple 
supply trays 410, allowing the printer to accommodate different sizes and types of paper 
as well as trays 410 of varying capacity. When the printer 100 needs blank paper for 
printing, the media supply handler 405 provides the print engine 420 with a sheet of blank 
medium. 
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[0039] The formatter 415 converts data received from the multimedia processing 
system 125 into a format that the print engine 420 can use to create an image on the 
paper. The print engine 420 creates an image on the paper as indicated by the formatter 
415. A flxser 425 then uses high temperature and pressure to fuse the image onto the 
paper to fix the image thereon. Once the image is fixed, the paper is fed to the media 
output handler 430. Although not shown, it is appreciated that the printer 100 includes 
any necessary motors, gears, and diverters to cause the paper to move through the printer 
100. 

[0040] The media output handler 430 receives one or more printed sheets of paper 
and performs any requested finishing to the sheets. For example, the media output 
handler 430 may include a sorter 435 to sort or collate the sheets for multiple copies and a 
stapler 440 to attach the sheets together. When the finishing process is complete, the 
media output handler 430 moves the sheets to an output 445, of which there may be 
multiple trays 445 to accommodate different sizes, types, and capacities of printed output. 

Electronic Output System 

[0041] The electronic output system 120 can be designed to produce an electronic 
output related to the multimedia data in any desired format. Because of the great variety 
of types and formats of electronic outputs, the electronic output system 120 may take any 
of a number of forms for producing an electronic output desired by the user. FIG. 5 
illustrates some examples of different embodiments of the electronic output system 105. 
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In particular implementations, the printer 100 may have only one or only a subset of the 
various components shown, and in addition it may have other types of not shown. 

[0042] In one embodiment, the printer 1 00 writes the electronic output to a 
removable media device with a media writer 505. Many different types of media writers 
are know in the art, and the media writer 505 many comprise any of these. For example, 
the media writer 505 may be configured to write the electronic output to removable 
storage devices such as a writeable DVD or CD, a video cassette tape, an audio cassette 
tape, a flash card, a computer disk, an SD disk, a memory stick, or any other appropriate 
electronically-readable medium. Moreover, the electronic output system 120 may include 
a number of media writers 505 of different types to allow the printer 100 to print onto 
different electronic formats. In addition, the electronic output system 120 may include a 
number of media writers 505 of the same type to increase the output capacity of the 
printer 100. 

[0043] The removable storage device that receives the electronic output from the 
printer 100 may be fed to the media writer directly by a user, for example by inserting a 
blank disk into a drive. In another embodiment, the printer 100 includes an electronic 
media handling mechanism 510 coupled to the media writer 505 that automatically 
provides the media writer 505 with an appropriate type of removable storage device. The 
handling mechanism 510 may further be configured to physically place written to storage 
devices into an output tray 515. In one embodiment, a series of blank storage devices are 
fed to the printer 100 by a bandolier 520 or other type of feeder, allowing the printer 100 
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to create a high volume of electronic output without requiring a significant amount of 
interaction with a human operator. The bandolier 520 preferably then places the written 
to devices into an output tray 515. 

[0044] In another embodiment, the media writer 505 is a disposable media writer, 
configured to write electronic data to a disposable removable media storage mechanism. 
In another embodiment, the media writer 505 writes the electronic data to a self- 
destructing medium. In this way, a user can view the electronic data for a predetermined 
number of times or during a predetermined period of time, after which the electronic data 
are no longer viewable. 

[0045] In another embodiment, the electronic output system 120 includes a speaker 
system 530. The speaker system 530 is designed to receive an audio signal from the 
media processing system 125, in response to which the audio is played from an embedded 
speaker 530 in the printer 100. The electronic output system 120 may further include a 
player 525 or audio renderer that receives an encoded audio signal from the media 
processing system 125 and converts it into an audio signal for the speaker 530. The 
player 525 thus takes some of the processing load off the media processing system 125. 
For example, the player 525 may include a MIDI player for generating the audio signal; 
however, many other audio Tenderers may be used, in either hardware or software. 

[0046] In another embodiment, the electronic output system 120 includes a video 
display 535. The video display 535 is designed to receive a video signal from the media 
processing system 125, in response to which the video is played on the video display 535 
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embedded into the printer 100. Similarly, the video display 535 may receive the video 
signal directly from a driver to reduce the processing load on the media processing system 
125. 



[0047] In another embodiment, the printer 100 transmits the electronic output that is 
to be printed to another device as a signal. This signal can later be fixed in a tangible 
medium by the external device. To facilitate this, the electronic output system 120 
includes a communication interface 540. The communication interface receives the 
electronic output from the media processing system 125 and sends the electronic output to 
the external device, which may be in communication with the printer 100 over a local 
network, the Internet, a wireless network, a direct connection, or any other suitable 
communication means. 

[0048] In another embodiment, the electronic output system 120 comprises an 
embedded web page display 545. The web page display 545 allows a user to see a 
representation of the electronic output in a web-based form. 

Multimedia Processing System 

[0049] The multimedia processing system 125 of the printer 100 is designed to 
perform the specialized functionality of the multifunction printer 100. To send and 
receive messages between external devices or the user interface 110, the processing 
system 125 includes a processor 135 and a memory 130. In addition, the multimedia 
processing system includes a multimedia module 140 that enables the printer 100 to 
create related printed and electronic outputs for different types of multimedia data. In this 
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way, the printer 100 can be configured to have any number of multimedia processing 
functionalities. 

[0050] In one embodiment, as illustrated in FIG. 6, multimedia module 140 
includes software and hardware to automatically detect 602 the coupling of a peripheral 
device 151, a module to communicate with a peripheral device 604, a module to process 
and format multimedia data 606, and a module to generate an output 608. 

[0051] In an embodiment, as illustrated in FIG. 7, the device detection module 602 
automatically detects (steps 702 and 704) the docking (or communicative coupling) of a 
peripheral device 151 and download multimedia data from the peripheral device for 
processing by the printer 100. The multimedia processor 125 can utilize such protocols 
as the Plug and Play (PnP) or Universal Plug and Play (UPnP) protocol to automatically 
detect and communicate with devices that has PnP capabilities. With UPnP, a device can 
automatically communicate with each other directly, convey its capabilities, and learn 
about the presence and capabilities of other devices. Those skilled in the art will 
recognize that other detection methods, such as polling techniques, may be used to 
automatically detect the coupling of a peripheral device. Alternative systems for 
connection, such as IEEE 1394 cabling or Universal Serial Bus cabling, have equivalent 
standards for device and capability discovery. Alternatively, one embodiment might 
detect the presence of an active electrical circuit in a physical connector, such as an 
RS232 serial port connector, between the docked device and printer 100. 
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[0052] Once a device is docked to the printer 100 via peripheral interface 105, the 
device communication module 604 is configured to communicate with the peripheral 
device 151. Again, PnP capabilities of a device will allow the communication module 
604 to communicate with the device. The communication module 604 may be configured 
with specific software drivers to enabling communication with specific peripheral 
devices. In an embodiment, the communication module 604 enables the printer 100 to 
operate a docked peripheral device to capture multimedia data. In such an embodiment, 
the printer 100 can also send commands to the peripheral docked peripheral device 151 
and operate the device by controlling certain functionalities of the device. For example, 
the printer 100 could communicate with a docked cell phone to automatically bill the 
owner of the cell through his/her cell phone provider, or based on information in an xml 
profile on the phone. In another example, the printer 100 can communicate with a docked 
cell phone to issue commands to take pictures with the cell phone's digital camera or to 
call a number from the cell phone. The picture might show who is standing at the printer 
100. That data could be transmitted to some other destination by the cell phone when it 
calls out. The called number could be provided in a profile stored on the cell phone, or it 
could be stored on the printer. These functions could be provided with an infrared 
interface that's available in many cell phones. Alternatively, the device discovery and 
capability negotiation on Bluetooth- or WIFI-enabled phones could identify the device 
and its capabilities to the printer. The printer would lookup the appropriate command set 
for that phone in a library of command sets stored on the printer or off-printer in a 
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network-accessible location. The printer 100 could then issue the commands necessary to 
invoke the desired functions. Other examples are discussed below. 

[0053] The communication module 604 sends a request to the peripheral device for 
multimedia data to be downloaded to the printer 100, step 706. Once the multimedia 
processing system 125 receives the multimedia data, the multimedia processing and 
formatting module 606 contains logic to format and process the multimedia data. In an 
embodiment, the format may be pre-defined and configured for specific peripheral 
devices. In other embodiments, a user may design a layout format for the multimedia 
data via the user interface 1 00. In yet another embodiment, a user may upload a 
predefined format. The multimedia processing and formatting module 606 provides an 
organized representation of the multimedia data depending on a particular peripheral 
device 151. For example, the formatter 606 may generate a format suitable for 
representation on a video paper. Thus, if a video camcorder is docked directly to the 
printer, the format might comprise key frames with bar codes linking the key frames to 
different segments of a video file that may be used to replay a recorded video describing 
the event. The formatter 606 can also create a format suitable for display on a web page. 
Examples of additional formats will be discussed below. 

[0054] In step 710, the multimedia output generator module 608 creates an output 
of the formatted multimedia data. The output may be sent to the printed output system 
115 where it will be printed on a document, or it may be sent to the electronic output 
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system 120 where an electronic output will be produced. In one embodiment, the output 
may also be uploaded to a web server via network 155. 

[0055] Although the media processing system 125 is configured to perform at least 
some of the processing of the media data on the printer 100, the system 125 can 
preferably share some of the computing burden with the peripheral device that 
communicatively coupled to it. In a preferred embodiment, the peripheral device 151 is 
capable of performing at least a portion of the multimedia processing functionality, such 
as formatting the multimedia data for output. In this way, the printer 100 is relieved of at 
least some of the processing burden required to implement the printer's functionality. 
The printer 100 can thus avoid slow-downs that can result from a heavy processing load, 
which may be especially important for shared printers. 

Interactive Communication with a Printer 

[0056] FIG. 8 shows an example of interactive communication with a printer in 
accordance with an embodiment of the present invention. 

[0057] In general, conventional printer drivers in modern operating systems are not 
designed to facilitate interactive information gathering. Because the print job can be 
redirected to another printer, or the printing protocol does not allow such interactive 
sessions, the operating system does not encourage interaction with the user. Once initial 
printer settings are captured, further interactions are generally not allowed in conventional 
printers. One approach to this problem is to embed metadata into the print stream itself, 
as noted above. However, it is possible that the printer could need to ask the user for 
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more information, in response to computations made from the data supplied by the user. 
In addition, the printer might itself delegate some tasks to other applications on other 
devices, which might in turn need more information from the user. So-called "Web 
services" or "grid computing" systems are examples of the sort of application server that 
the printer might trigger. 

[0058] In order to allow this interaction, without modifying printer driver 
architecture of the underlying operating system, an extra mechanism, such as the one 
shown in Fig. 8, is constructed. A "UI Listener," program 854 listens to a network 
socket, accepts requests for information 808, interacts with a user to obtain such data, and 
then sends the data back to the requester. 

[0059] Once a print request 802 is sent by user 850 and notification requested from 
the UI listener 804, the print job is sent by application 852. Here, the print job contains 
embedded information including the network address of the UI listener, authentication 
information, and the latest time that the client will be listening for requests. 

[0060] If the printer requires additional information of confirmation, it sends a 
request 808, which is detected by the UI listener, which displays a dialog box to obtain 
input from the user 810. An example of such a request might be a request for a password 
or user confirmation code that the user must enter to access a database 858. The user's 
input is included in a reply 812 sent to the printer. If the reply does not satisfy the printer 
it may ask for additional information (not shown). If the reply does satisfy the printer, it 
takes a next step. This step might be to perform an external action such as sending an 
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email (not shown). The next step might also be sending a request for information 814 to 
an application server (such as a database) 858. In this example, application server 858 
also sends a request for information 816, which is detected by the UI listener. The user is 
prompted 818 and his response forwarded to the application server 820. In this example, 
a reply is then sent from the application server 858 to the printer 856. It will be 
understood that a particular embodiment may include either or none or requests 808 and 
816 without departing from the spirit of the present invention. 

[0061] A program such as that shown in Fig. 8 may have a fixed set of possible 
interactions, or may accept a flexible command syntax that allows the requester to display 
many different requests. An example of such a command syntax would be the standard 
web browser's ability to display HTML forms. These forms are generated by a remote 
server, and displayed by the browser, which then returns results to the server. In this 
embodiment, however, the UI listener is different from a browser in that a user does not 
generate the initial request to see a form. Instead, the remote machine generates this 
request. In the described embodiment, the UI listener is a server, not a client. 

[0062] Because network transactions of this type are prone to many complex error 
conditions, a system of timeouts would be necessary to assure robust operation. 
Normally, each message sent across a network either expects a reply or is a one-way 
message. Messages which expect replies generally have a timeout, a limited period of 
time during which it is acceptable for the reply to arrive. In this embodiment, embedded 
metadata would include metadata about a UI listener that will accept requests for further 
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information. Such metadata preferably includes at least a network address, port number, 
and a timeout period. It might also include authentication information, designed to 
prevent malicious attempts to elicit information from the user. Because the user cannot 
tell whether the request is coming from a printer, a delegated server, or a malicious agent, 
prudence suggests strong authentication by the UI listener. If the printer or a delegated 
application server wishes more information, it can use the above noted information to 
request that the UI listener ask a user for the needed information. 

Printer with Cell Phone Interface 

[0063] A cell phone is docked directly with the printer 1 00, thereby eliminating the 
need to attach the cell phone to a PC. The multimedia processor 125 of the printer 100 
automatically downloads still frame images, audio and video clips that are resident on the 
cell phone to the printer 100. The multimedia processing and formatting module 606 
formats the data as a video paper document, and prints that document. The multimedia 
data output can also be sent to an electronic output system 120 where the data can be 
written on an external storage site, or a removable media card. The media card can be 
played on the cell phone using the cell phone digital camera as a bar code reader. Or if 
the media were uploaded to a web site via network connection 155 that provides a 
streaming media service, the user could call that site and play the media on the cell phone 
after entering the necessary access codes. Keying digits on the keypad of the cellular 
phone provides random access to points in the recording indicated by bar codes and other 
human-readable text. That text can also include phone numbers called while the 
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corresponding multimedia data were recorded. If those phone numbers correspond to 
address book data on the cellular phone, it can also be included in the print out 165. 

Printer with Hardware and Software Interface for Camcorder 
[0064] A video camcorder is docked directly with the printer 100, thereby 
eliminating the need to attach the camcorder to a PC, and the still frame images, audio 
and video clips that are resident on the camcorder are downloaded to the printer 100 via 
communication module 604 on the multimedia processing module 140, which formats the 
data as a video paper document, and prints that document. The multimedia data can also 
be written on an external storage site, or a removable media card. The media card can be 
played on the camcorder, using its camera as a bar code reader. Or if the media can be 
uploaded to a web site that provides a streaming media service via network interface 150 
that is couples the printer to a web server on the network 155; the user can login to that 
site later and play the media in a web browser after entering the necessary access codes. 
Other human-readable text could be printed near the bar codes that indicate the data and 
time when the recording was made as well as the location of the person, if available from 
something like a GPS receiver. The printer 100 could also be configured to operate the 
camcorder and send it commands that cause it to capture video and audio of the 
environment nearby the printer. The user could also be provided instructions over user 
interface 1 10a that tell her how the device should be manipulated at any time. For 
example, where to point the camcorder. 

Printer with Hardware and Software Interface for Digital Audio Recorder 
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[0065] An external digital audio recorder (DAR), like those used by doctors to 
record notes about patients' medical charts, is interfaced to the printer 100. The display 
on the printer 100 is configured to read information about the content of the DVR over a 
serial port. This information is printed in various formats, such as a program guide or as 
an Audio Paper document that shows the meta information about each recording such as 
when it occurred, its duration, where the user was supposed to be, as provided by his 
online scheduler, etc. Bar codes, when swiped, cause the DAR to begin playing the audio 
recording from the corresponding point. Those clips can also be uploaded to a web site 
that provides access to the audio over the phone. The user can call the audio server and 
key in access code information as well as clip identification data that cause any clip to 
begin playing from the appropriate point in time. The printer 100 could also be 
configured to operate the DAR and send it commands that cause it to capture audio from 
the environment nearby the printer. The user could also be provided instructions over 
user interface 1 10a that tell her how the device should be manipulated at any time. For 
example, where to point the DAR or what to say to it (e.g., "Pronounce your name"). 

Printer with Hardware and Software Interface for Portable Meeting Recorder 
[0066] A portable meeting recorder, composed of a video camera with a 360-degree 
lens and a microphone, is docked to the printer 100. The multimedia data files are 
downloaded to the printer. The multimedia processing and formatting module 140 
dewarps the video and extracts the key frames from the multimedia data files. Those key 
frames are chosen to show at least one image of every meeting attendee. Face detection 
and tracking can be used to find the largest and most well-composed image of each 
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participant. A video paper document (printed output 165) is created from the multimedia 
recording that includes bar codes that refer to the multimedia data stored on the Portable 
Meeting Recorder, or an associated PDA. The multimedia is played by swiping the bar 
codes with a tethered bar code reader or by imaging the bar codes with a video camera 
interfaced to the Portable Meeting Recorder. The printer 100 could also be configured to 
operate the Portable Meeting Recorder and send it commands that cause it to capture 
audio and video from the environment nearby the printer. The user could also be 
provided instructions over user interface 1 10a that tell her how the device should be 
manipulated at any time. For example, where to point the Portable Meeting Recorder or 
what to say to it (e.g., "Pronounce your name"). 

Printer with Hardware and Software Interface for Fixed Position Meeting Recorder 

[0067] A meeting recorder in a conference room is equipped with a Print button. 
Pressing the Print button triggers software that formats the most recently recorded 
meeting as Video Paper with bar codes that refer to data stored on the local system. The 
appropriate audio capture software and hardware are provided so that sound localization 
can be performed in real-time and post-processed to produce Video Paper which 
segments the recording by the position from which the sound emanated in the room. A 
suitable representation is used to provide access to the recorded data. For example, 
sectors of a circle can be annotated with colors and bar codes that provide access to the 
points in the recording when the person at that location began speaking. Additional meta 
information could be printed next to the sector that shows how many times the person at 
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that location spoke and the total amount of time they used. This could also be encoded in 
the color assigned to the sector. For example, red could indicate the person who 
dominated the meeting. A similar sector icon could be used to annotate a time line so 
that by looking at the time line and the icon, the user could tell where the person was 
sitting who was speaking at that time and approximately how long they spoke for. 

Printer with Hardware and Software Interface for Fixed Position Meeting Recorder with 
Pan Tilt Zoom Capture Capability 

[0068] A meeting recorder camera in a conference room has the ability to pan, tilt, 
and zoom its view while recording a meeting. These capabilities can be combined with 
real-time sound localization and tracking to produce a video recording which captures not 
only the dialog in a meeting but also what is written on a whiteboard and what appears on 
the display screen for the projector. This system can also be equipped with a Print button 
that when pressed generates a Video Paper representation for the recorded multimedia. 
This document will include images of the whiteboards when their content changes 
significantly as well as key frames that were captured when the camera head was panned 
or tilted significantly to track action in the room. 

Printer with Hardware and Software Interface for Head-Mounted Video Camera with 
Gyroscope and GPS Capture 

[0069] A head-mounted video camera, such as those commonly used to capture the 
video record of a Cyborg, is docked with the printer 100. It downloads the multimedia 
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data from the camera and creates a video paper representation for it that is outputted to 
the electronic output system 120 and stored on the printer 100 or reprocessed and 
uploaded to a web hosting service via network 1 55. The multimedia processing system 
125 extracts key frames the video file when the person's head moves significantly, as 
detected by the gyroscopic tracking software in the camera, or when the person's 
geographic position changes significantly, as indicated by the GPS system. This data can 
be correlated with symbolic information stored on the web or in the user's personal 
organizer which correlates geographic position with logical location, such as 36 degree 20 
minutes North 47 degrees 2 minutes West is Work, to produce a document that can be 
selectively formatted to show only the people spoken with at Work, for example. In 
another case, the individual might choose to print only the faces of the people he met 
while at Costco. The printer 100 could also be configured to operate the Head-Mounted 
Video Camera and send it commands that cause it to capture audio and video from the 
environment nearby the printer. The user could also be provided instructions over user 
interface 1 10a that tell her how the device should be manipulated at any time. For 
example, where to point the camera or what to say to it (e.g., "Pronounce your name"). 

Printer with Hardware and Software Interface for Office-based PC Experience Capture 
System 

[0070] One or more video cameras are configured to record the actions that take 
place in a user's office during the normal workday. This system can distinguish between 
a person working in isolation at their PC and new people walking into the office. This 
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system is equipped with a Print button that when pressed creates a Video Paper document 
which shows key frames and bar codes representative of events that the system has been 
pre-configured to consider interesting. For example, this might show a key frame and bar 
code for every visitor that entered the office that day. The printer 100 could also be 
configured to operate the Office-based PC Experience capture system and send it 
commands that cause it to capture audio and video from the environment nearby the 
printer. The user could also be provided instructions over user interface 1 10a that tell her 
how the device should be manipulated at any time. For example, what to display on the 
PC screen (e.g., "show me your schedule for today") where to point the video camera or 
what to say to it (e.g., "Pronounce your name"). 

Printer with Hardware and Software Interface for Video from PC Frame Buffer 
[0071] The video that appears on a PC user's screen is captured using the video 
capture 355 by sampling key frames from the PC's display buffer and creating a video file 
from that, or by tapping the video cable from the PC to its monitor. A video paper 
document is created from that which shows key frames only when there is a significant 
change in what's displayed on the monitor of the PC. This can occur when the user views 
a new web page, for example. Bar codes are displayed near those key frames that provide 
access to the video file. The printer 100 could also be configured to operate the PC frame 
buffer capture system and send it commands that cause it to capture frames. The user 
could also be provided instructions over user interface 1 10a that tell her how the device 
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should be manipulated at any time. For example, what should be displayed on the PC 
screen. 

[0072] While examples of suitable printing systems are described above, the 
description of the printer and its document production means is not meant to be limiting. 
Depending on the intended application, a printer can take many different forms other than 
the typical office or home-use printer with which most people are familiar. Therefore, it 
should be understood that the definition of a printer includes any device that is capable of 
producing an image, words, or any other markings on a surface. Although printing on 
paper is discussed above, it should be understood that a printer in accordance with 
various embodiments of the present invention could produce an image, words, or other 
markings onto a variety of tangible media, such as transparency sheets for overhead 
projectors, film, slides, canvass, glass, stickers, or any other medium that accepts such 
markings. 

[0073] In addition, the description and use of multimedia and multimedia data are 
not meant to be limiting, as multimedia may include any information used to represent 
any kind of media or multimedia content, such as all or part of an audio and/or video file, 
a data stream having multimedia content, or a transmission of multimedia content. 
Multimedia content may include one or a combination of audio (including music, radio 
broadcasts, recordings, advertisements, etc.), video (including movies, video clips, 
television broadcasts, advertisements, etc.), software (including video games, multimedia 
programs, graphics software, etc.), and pictures; however, this listing is not exhaustive. 
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Furthermore, multimedia data may farther include anything that itself comprises 
multimedia content or multimedia data, in whole or in part, and multimedia data includes 
data that describes a real-world event. Multimedia data can be encoded using any 
encoding technology, such as MPEG in the case of video and MP3 in the case of audio. 
They may also be encrypted to protect their content using an encryption algorithm, such 
as DES, triple DES, or any other suitable encryption technique. 

[0074] Moreover, any of the steps, operations, or processes described herein can be 
performed or implemented with one or more software modules or hardware modules, 
alone or in combination with other devices. It should farther be understood that portions 
of the printer described in terms of hardware elements may be implemented with 
software, and that software elements may be implemented with hardware, such as hard- 
coded into a dedicated circuit. In one embodiment, a software module is implemented 
with a computer program product comprising a computer-readable medium containing 
computer program code, which can be executed by a computer processor for performing 
the steps, operations, or processes described herein. 

[0075] In alternative embodiments, the printer can use multiple application servers, 
acting in cooperation. Any of the requests or messages sent or received by the printer can 
be sent across a network, using local cables such as IEEE 1394, Universal Serial Bus, 
using wireless networks such as IEEE 802.1 1 or IEEE 802.15 networks, or in any 
combination of the above. 
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[0076] The foregoing description of the embodiments of the invention has been 
presented for the purpose of illustration; it is not intended to be exhaustive or to limit the 
invention to the precise forms disclosed. Persons skilled in the relevant art can appreciate 
that many modifications and variations are possible in light of the above teachings. It is 
therefore intended that the scope of the invention be limited not by this detailed 
description, but rather by the claims appended hereto. 
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